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(54) Title: HANDWRITTEN CHARACTER RECORDING AND RECOGNITION DEVICE 



^^^^ (57) Abstract: The invention is an electronic recording and computing device that 

1 y, resides within or on a pen shaped object for the purpose of recording and processing 

handwritten text or graphics. The device includes a marking implement that may be 
a pencil, ink pen, non-maiking tip, chalk, crayon, marker, paint, laser maiking tool, 
^ or any other instrument used for writing or marking. The device records motion and 
marking with one or more of each of the following components: image sensor (1), 
2 light source (5), light detector, force sensor (3), contactor, microprocessor, volatile and 
non-volatile memory, analog to digital converter. Movement of the apparatus is sensed 
and recorded by means of the image sensor (1) and optica] navigation software which 
tracks microscopic and/or macroscopic features of the writing surface. The orientation 
of the pen to the writing surface is electronically sensed and recorded by casting a plane 
wave of light on the surface and detecting the amount received back with several light 
sensors placed in a specific orientation with each other. 



wo 02/069247 



PCT/USOl/05689 



DESCRIPTION 

TITLE 

HANDWRITTEN CHARACTER RECORDING AND RECOGNITION DEVICE. 
FIELD QF INVENTION 

The invention relates to handwritten character input and recognition devices. More particularly the device 
provides a means to electronically record handwritten graphics and record and recognize hand written 
characters. 

BACKGROUND ART 

There are numerous patents related to this type of device. The distinguishing features lie in the method or 
methods used to sense the movement of the device to the writing or marking surface. All such devices 
employ a button, contactor, or force sensor to sense when the apparatus is in contact with the writing 
surface. The combination of the above data can be used to record the movement and marking of the pen. 

One method attempts to measure forces applied to the pen tip or pen cartridge by using strain gauges, force 
sensors, piezo-electric devices, or the like (US4751741, US5215397). These forces are used to calculate 
the movement of the device across the page. 

Another method uses accelerometers or gyroscopes to sense acceleration and deceleration of the pen 
(US188392.US5247137,US5517579). 

An optical technique is used to image a known pattern on the page and use this information to calculate 
movement of the apparatus in (US5226091). 

Finally there are also approaches that use a combination of the above sensing techniques. These include a 
combination of force and optical sensing (US5215397) and a combination of acceleration and ultrasonics 
(US55 17579). 

THE PRESENT INVENTION 

The invention is an electronic recording and computing device that resides within or on a pen shaped object 
for the purpose of recording and processing handwritten text or graphics. The device includes a marking 
implement that may be a pencil, ink pen, non-marking tip, chalk, crayon, marker, paint, laser marking tool, 
or any other instrument used for writing or marking. The device records motion and marking with one or 
more of each of the following components: image sensor, light source, light detector, force sensor, 
contactor, microprocessor, volatile and non-volatile memory, analog to digital converter. Movement of the 
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apparatus is sensed and recorded by means of the image sensor and optical navigation software which 
tracks microscopic and/or macroscopic features of the writing surface. The orientation of the pen to the 
writing surface is electronically sensed and recorded by casting a plane wave of light on the surface and 
detecting the amount received back with several light sensors placed in a specific orientation with each 
other. This information is used to correct optical navigation data that is distorted by the orientation of the 
pen to the writing surface. Data may be processed by on board application software and/or stored for later 
retrieval. Data may be transmitted through a docking station or via a wireless technology. The transmitted 
information may later be used with software for tasks such as optical character recognition, signature 
authentication, or image reconstruction. 

It is an object of the invention to eliminate the need for expensive and inaccurate accelerometers or 
gyroscopes, to eliminate unreliable and inaccurate mechanical systems, and to use inexpensive off the shelf 
electronic components. It is furthermore an object of the present invention to rely primarily on optical 
methods for sensing movement and orientation of the device to the marking surface because these well 
developed technologies allow for a high level of accuracy and repeatability. 



DESCRIPTION OF FIGURES 

1. An embodiment of the device including 1 image sensor, 2 fiber optic image pipe, 3 button or force 
sensor, 4 writing implement cartridge and force sensor actuator, 5 light source and light sensor array, 
and 6 telocentric lens system. 

2. A possible arrangement of light sensors (filled) and light sources (not-filled) used for sensing 
orientation of the apparatus relative to the marking surface. 

3. A plane wave of light may be created with an array of light sources thus providing consistent light 
intensity across a large area. 



FIRST EMBQDIMKNT 

A first embodiment of the device is shown in FIG 1 and is constructed in accordance with claims of the 
present invention. A pen shaped enclosure provides a housing for all components of the system such that 
the apparatus may operate autonomously. 

The embodiment also comprises a marking implement for writing characters or making graphical markings 
on paper or other surface. The marking implement in this first embodiment is ink. The marking implement 
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resides within a cartridge such that the cartridge may move within the pen so that it may engage a force 
sensor for detecting when the tip of the device is in contact with the marking surface (FIG. 1-3) 

This embodiment is also comprised of one or more infrared light emining diodes arranged in a manner to 
produce a plane wave of light which is to be incident on the marking surface.(FZG 1-5) 

The embodiment is also comprised of three infrared phototransistors for detecting infrared light from said 
light emitting diodes that has reflected off the marking surface. The phototransistors are arranged in a 
specific orienuiion to each other in this embodiment of the device(FIG 2). The amount of infrared light 
detected is used to calculate the distance and or angle of inclination of the apparatus to the marking surface. 
Using data obtained from two light sensors the angle of inclination of the apparatus relative to the writing 
surfoce may be calculated in two orthogonal directions. 

The embodiment is also comprised of an imaging system that includes a charged coupled device image 
sensor, two lenses, and a fiber optic image pipe(FIGl-l^,6). The two lenses are chosen such that the 
image obuined by the imaging system provides a telecentric view of the page, this ensures there is little or 
no distortion in the image. This imaging system is used to rapidly obtain images of macroscopic and/or 
microscopic features of the marking surface. These images are used in conjunction with optical navigation 
software capable of comparing these images and mathematically calculating the motion of the apparatus 
relative to the marking surface. 

The embodiment also consists of a microprocessor, memory, and application software that includes said 
optical navigation as well as software necessary for processing data obtained from said phototransistors. 
This information may be stored in memory for later retrieval or transmitted in real time to a personal 
computer or other computing device. 

The embodiment of the device also includes a display and several buttons so the user may interact with the 
device and invoke application software or data transmission. 

This embodiment of the device includes electronic hardware and/or software to support the transmission of 
data via a wired or wireless means. 
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CLAIMS 

1. An apparatus capable of electronically recording handwriting or marking of a surface and/or 
electronically recording and recognizing handwritten characters and comprising of the following: 

• an imaging system 

• a sensor capable of detecting when the writing or marking tip of the device is in contact with a 
surface 

• a light source 

• a light detector or light detectors 

• data processing means 

• data acquisition means capable of obtaining signals from said light detector or light detectors for 
use by said data processing means 

• a writing or marking implement 

2. An apparatus according to claim 1 where said imaging system is comprised of one or more of the 
following: a solid state imaging device which may be a charged coupled device or CMOS image 
sensor, a lens or series of lenses, and a fiber optic image pipe.(FIG. 1) 

3. An apparatus according to claim 2 where said lens or series of lenses comprises a telecentric lais 
system. Said telecentric lens system is defined here as a lens system that delivers an image of constant 
magnification with no perspective error when the distance to the objective changes.(FIG. 1-6) 

4. An apparatus according to claim 2 capable of obtaining images of microscopic and/or macroscopic 
features of the writing or marking surface using said image sensor. These features may be inherent to 
the material or be writing or markings created by the invention. 

5. An apparatus according to claim 1 where said sensor capable of detecting when the writing tip of the 
apparatus is in contact with a surface is a simple contactor or is a force sensor also capable of sensing 
how much force is being applied to the apparatus against the surface (FIGl-3). 

6. An apparatus accordmg to claim 1 where said data processing means are capable of analyzing images 
obtained from said imaging system for the purpose of calculating movement of the apparatus relative 
to the marking surface. 

7. An apparatus according to claim 1 where said light source is comprised of one or more light emitting 
elements and where said light source is capable of producing a plane wave of light(FIG. 3). A plane 
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wave of light is defined in this context as having equal or nearly equal amplitude across the emitted 
beam or a poition of the emitted beam. 

8. An apparatus according to claim 1 where a filter allows only a particular wavelength of light to be 
detected by said light detectors. 

9. An apparatus according to claim 1 where said light detectors are arranged in a manner where they may 
be used to measure the distance of the light detector to the surface based on the amount of light 
detected. Said light is light originating from said light source that has reflected off the marking 
surface. Data obtained from said light is used to calculate the orientation of the plane in which the 
light detectors reside to the plane of the writing surface. 

10. An apparatus according to claim 1 where said data processing means are capable of calculating the 
angle of inclination in one or more directions and/or the distance to the marking surface based on the 
amount of light detected by said light detectors. 

1 1. An i^yparatus according to claim 1 capable of wired and/or wireless transmission of data. 

12. An apparatus according to claim 1 1 capable of sending data directly to a printing device. 

13. An apparatus according to claim 1 also comprising an electronic display for providing a user interface 
or displaying data. 

14. An apparatus according to claim 1 also comprising application software that may includCt but is not 
limited to character recognition, email, web browsing, calculator, networking, and function graphing. 

15. An apparatus according to claim 1 also comprising one or more buttons for use as a user interface to 
application software. 

16. An apparatus according to claim 1 also comprising one or more lights or light sources for use as 
indicators to the state of operation of the device. 

17. An apparatus according to claim 1 also comprising an electronic system capable of reading barcodes or 
similar encoded information. 
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Figure 1 
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Figure 2 
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